As started in part 1 of this series, I have set out to test an LLMs ability to technical edit. For my first set of tests, I am using a pair of articles I created, filled with very bad advice. The advice is the same for both articles, but what differs is the intro and the conclusion. One says the advice is good, the other said it is bad. It is all very very bad, including a really terrible SELECT statement versus loop construct that will cause an eternal loop that inserts into a temporary table.
My goal is to see how much of that advice will be noted as bad, and if it says anything nice at all about the text, etc. If you want to see the entire documents, you can get them here in a zip file, both in text and word document formats.
If you are questioning this concept of using very poorly written source material, that is understandable. After I finished this article’s first pass, I realized that if I was going to really do a good job testing these models, I am going to need to test with more subtle incorrectness mixed in with correct statements. However, I feel as if there is something to learn from this first experiments which are chronicled here in part 2 and will be with part 3 as well, before I change gears.
Before I get started, I asked ChatGPT “what model of ChatGPT am I using now?” just to show you where it is at. The reply was:
“You’re currently using ChatGPT-4 with additional features enabled, such as code execution, web browsing, and a document editor for enhanced interactivity and productivity.”
So whatever that meant on February 1, 2025, it is what it is.“You’re currently using ChatGPT-4 with additional features enabled, such as code execution, web browsing, and a document editor for enhanced interactivity and productivity.”
Getting started with ChatGPT
Okay, I have not used ChatGPT on this document yet at all. I do not have a paid account, so I will be using the free version, with a login. I will try two tests. First, the entire documents will be reviewed and I will give the highlights. Then I will take a section and dig into whatever feedback it gives me.
The phrase I have found works with Copilot quite well to get it to only give me technology issues has been: “Please review the following document in terms of technology”. So I will be using this as my starting point, though I may change it up a little here and there along the way.
Disclosure: as I originally typed this introduction, I had not used ChatGPT for any work with this document at all, nor had I used Copilot. I just created the documents and edited them with Grammarly.
Document advocating for the advice
So here we go. I typed:
“Please review the following document in terms of technology”
And then pasted the document with the header that said this was good advice. The result, was oddly polite, but correct. It starts out:
“This document contains several practices and recommendations for SQL Server administrators. However, many of them reflect misconceptions, outdated practices, or outright harmful advice in the realm of modern SQL Server administration. Below is a review of each suggestion from a technical standpoint.”
I liked that it didn’t put any positive or negative in the first sentence. It just said “it contains some stuff.” Kind of like a person might when trying to say ANYTHING positive about an article that is such garbage.
It ends with:
“While some advice here might be well-intentioned, much of it is outdated, misleading, or counterproductive in modern SQL Server environments. Following these suggestions could lead to security vulnerabilities, performance issues, and operational inefficiencies.”
Well-intentioned. It is almost like it’s trying to be too kind here. It goes on to offer to rewrite the document for me. I was not expecting this, but that will be my next thing to try. I saved the full text of the results here: results if you wish to review.
The main thing is I am not getting any deeper information. For example, one section in my document contains the following incomprehensibly wrong content:
“- SQL Server works better with LOOPING code
If you want to filter from a set of data, the best way is to use a WHILE WEND construct and do the filtering ROW BY ROW. For example, say you have a table with a million rows. You can write:
SELECT *
FROM TableName
WHERE Criteria = False;
But it is often far faster to use a loop such as:
CREATE TEMP TABLE #Results
WHILE (RowsStillInTable)
BEGIN
INSERT INTO #Results
SELECT TOP 1 *
FROM TableName
WHERE Criteria = False;
IF @@RowsCount = 0
Break;
WEND;
By using the TOP 1 modifier, this allows each iteration to execute much faster than the SELECT statement that needs to touch every row in the table simultaneously.”
“SQL Server Works Better with Looping Code
Review: Iterative row-by-row processing (RBAR—”Row By Agonizing Row”) is inefficient. Set-based operations are the foundation of SQL and are far more performant.
Recommendation: Use set-based SQL queries and avoid looping constructs whenever possible.”
And the reply says:
“SQL Server Works Better with Looping Code
Review: Iterative row-by-row processing (RBAR—”Row By Agonizing Row”) is inefficient. Set-based operations are the foundation of SQL and are far more performant.
Recommendation: Use set-based SQL queries and avoid looping constructs whenever possible.”
Reading over that advice, you should see the code itself is completely wrong. After I let it do a rewrite, I am going to ask it to tell me more about what is wrong in that section.
Let’s see if it can rewrite this for me?
Next, I say yes to letting it rewrite the article for me. You can easily tell that the output it is AI generated, if for no other reason than the overuse of bullet points. The tips come out in a very basic manner, but the tips are generally ok. The document is basically lifeless and example-less, but correct. The replacement for my terrible example code that had code:
“12. Avoid Row-by-Row Processing
Row-by-row processing (RBAR) is inefficient. SQL Server is optimized for set-based operations. To enhance performance:
- Use set-based queries whenever possible.
- Rewrite RBAR logic into efficient set-based T-SQL statements.”
RBAR? I don’t love that term to start with, but it does not stand for Row-by-row processing, it stands for Row By Agonizing Row. I don’t love it because sometimes you do need to loop. And it makes it sound too much like a human doing work. (Or maybe because I didn’t come up with it, who knows.
The biggest issue is that it didn’t even try to understand my code, or anything of the sort. It simply gave a bit of advice.
The complete rewrite results can be found here.
Digging into the advice
Finally, I asked it:
“Give me advice on this section alone:”
And pasted in the section 12 as it was shown earlier, with all the really terrible code. The reply:
“Issues with the Given Advice
- Inefficiency of Row-by-Row Processing:
- Using a loop to process rows one at a time introduces significant overhead.
- Each iteration involves separate queries, context switches, and transactions, which degrade performance.
Misconception About SELECT Efficiency:
- A SELECT query with proper indexing can filter data efficiently without requiring loops.
- The suggestion to use TOP 1 in a loop does not improve performance for large datasets; it just splits the work inefficiently.
Unnecessary Complexity:
- The loop adds complexity to the code and increases the likelihood of bugs.
- Temporary tables like #Results are not needed for filtering data in this scenario.
Correct Approach: Use Set-Based Queries
SQL Server is built to handle set-based operations efficiently. To filter data based on a condition, use a single SELECT statement with appropriate indexing:
SELECT *
FROM TableName
WHERE Criteria = 0; — Assuming Criteria is a binary or Boolean column
Optimizations for Large Datasets
If performance is a concern for large datasets, consider these optimizations:
Indexing:
Ensure there is an index on the Criteria column to speed up filtering.
Partitioning:
Partition large tables to distribute data and improve query performance.
Query Hints:
Use query hints like OPTION(RECOMPILE) if parameter sniffing is an issue.
Batch Processing:
If you need to process the data incrementally, use batch processing instead of row-by-row loops:
WHILE EXISTS (SELECT 1 FROM TableName WHERE Criteria = 0)
BEGIN
DELETE TOP (1000)
FROM TableName
WHERE Criteria = 0;
END;
Summary
- Avoid loops for filtering data in SQL Server; use set-based queries instead.
- Optimize performance with indexing, partitioning, and batch processing if necessary.
- Row-by-row processing should be a last resort for edge cases where set-based logic cannot be applied.
This approach aligns with SQL Server’s strengths and ensures better performance, simplicity, and maintainability.”
Ok, so it is clearly better than my advice, but really, it ignored the meat of my poor code. The advice is more or less correct (some assumptions like Criteria is a bit or integer value), but also subtly different than other advice it had given me, which is always a concern because why would it give different advice for the same input?
For example, it goes on to talk about deletes for some reason, when I was only talking about a SELECT statement.
Explaining my cruddy code
Last thing I will try is to just ask it to explain my terrible code. So I put in the following prompt:
“Explain this code:
CREATE TEMP TABLE #Results
WHILE (RowsStillInTable)
BEGIN
INSERT INTO #Results
SELECT TOP 1 *
FROM TableName
WHERE Criteria = False;
IF @@RowsCount = 0
Break;
WEND; “
This gave quite a reply. Let’s look at the replies:
“Creating a Temporary Table
CREATE TEMP TABLE #Results
This line attempts to create a temporary table named #Results. However, there is a syntax error here:
In SQL Server, the correct syntax is CREATE TABLE #Results, not CREATE TEMP TABLE.
Temporary tables (prefixed with #) exist only for the duration of the session or batch in which they are created. They are often used to store intermediate results.”
Yep, guilty! However, I am also guilty of not providing columns or returning the results in the loop, which it failed to note. One thing I am starting to note is that it doesn’t seem to really dig into multiple facets of incorrectness, but if it gives one that is really great.
Continuing into the reply:
“The WHILE Loop
WHILE (RowsStillInTable)
BEGIN
This is the start of a WHILE loop that is supposed to run as long as the condition (RowsStillInTable) is true.
However, RowsStillInTable is not a valid SQL Server construct, and this part of the code is incorrect or incomplete. Typically, the condition would need to be a Boolean expression, such as checking if ows still exist in the source table.”
Something to notice here. Be careful if you are using this to look at something that includes pseudocode (especially if it isn’t labeled as such. It doesn’t realize that this could be just saying to check for rows in the table. I didn’t enumerate that, so this is good stuff too.
Sometimes I find it taking RowsStillInTable as a Boolean value (which SQL Server doesn’t support), or as an integer, but never did it consider that it might just be pseudocode.
“3. Inserting Rows Into the Temporary Table
INSERT INTO #Results
SELECT TOP 1 *
FROM TableName
WHERE Criteria = False;
This statement inserts a single row (TOP 1) from TableName into the temporary table #Results where the Criteria column has the value False (or 0, depending on data type).
The TOP 1 modifier limits the result to just one row, but without an ORDER BY clause, the row returned is arbitrary and not deterministic.”
Here is the first really big failure in checking this code. “where the Criteria column has the value False (or 0, depending on data type)” there is no way in SQL Server for the value to have a value False without quotes. ‘False’ can be translated to 0 in some cases, but not typically like this. This will work:
|
1 2 3 4 |
SELECT CASE WHEN CAST(1 as bit) ='True' THEN 1 ELSE 0 END, CASE WHEN CAST(0 as bit) ='True' THEN 1 ELSE 0 END, CASE WHEN CAST(1 as bit) ='False' THEN 1 ELSE 0 END, CASE WHEN CAST(0 as bit) ='False' THEN 1 ELSE 0 END |
And return 1, 0, 0, 1, respectively, but remove the quotes around True or False and you will get an error:
Msg 207, Level 16, State 1, Line 4Invalid column name 'False'.
Continuing into the reply:
“Checking the Number of Rows Inserted
IF @@RowsCount = 0
Break;
@@ROWCOUNT is a system function that returns the number of rows affected by the last statement.
Here, it checks if the previous INSERT INTO operation inserted zero rows. If no rows were inserted, the loop exits using the BREAK statement.”
Hey something that works correctly as I wrote it. Who saw that coming? That won’t last long.
“End of the WHILE Loop
WEND;
WEND is not valid SQL Server syntax. The correct keyword to close a WHILE loop in SQL Server is END.
The corrected version would look like this:
WHILE (condition)
BEGIN
— Loop body
END;”
Interestingly, IT wrote pseudocode by using (condition), but never considered my pseudocode…
“Key Issues in the Code
Syntax Errors:
CREATE TEMP TABLE should be CREATE TABLE #Results.
WEND should be replaced with END.
Inefficiency:
This code processes rows one at a time, which is inefficient for large datasets.
A SELECT statement with appropriate filtering is more efficient.
Lack of Determinism:
The TOP 1 query lacks an ORDER BY clause, which may lead to inconsistent results.
Unclear Loop Condition:
The condition (RowsStillInTable) is undefined and would not work in SQL Server.”
Mostly true stuff. I don’t think TOP 1 with an ORDER BY technically provides determinism exactly, as data can change while you are looping. You would need to use a temporary table or even a static cursor for that.
But the description mostly makes sense. And I will give it a pass since I didn’t provide a reasonable use case for using a loop here.
Finally, some advice, but with a bit of a mistake:
“A Correct and Efficient Approach
If the goal is to process or filter rows from a table, use a set-based query instead of a loop:
SELECT *
INTO #Results
FROM TableName
WHERE Criteria = 0;
This query filters all rows meeting the condition (Criteria = 0) and inserts them into the temporary table in a single operation.”
But if you send them to a temp table, this query basically does nothing. Especially when I said this loop did the same thing as my simple SELECT statement.
It also takes my pseudocode of = False and assumes I am using a bit, which is a more forgivable rewrite that I may have done myself.
And the final advice:
” Conclusion
The provided code is syntactically and conceptually flawed for SQL Server. To process data efficiently:
Use set-based operations wherever possible.
Avoid row-by-row logic unless absolutely necessary.
Fix syntax errors like CREATE TEMP TABLE and WEND.”
Generally, makes sense, except for the fact that I asked: “Explain this code”, not “Tell me what is wrong with it philosophically”
What it needed to note was something such as:
“This code appears to be trying to load data into a temporary table #Results. There may be a better way to do this using the following code:
SELECT *
INTO #Results
FROM TableName
WHERE Criteria = 0;
Then perhaps break down the code, tell me the syntax issues, and preach at me that row by row logic is bad.
Document Telling Us We are looking at bad advice
My last test was to basically admit that the body of the article was completely garbage. I changed it into a “here is bad advice to not follow” kind of article. So I altered the document to start out as:
“I asked many SQL Server administrators for the absolute dumbest advice they have ever been given by real people recently and this (perhaps a bit too humorous) article is the culmination of what I was told. The advice ranges from horrible to the extremely horrible, but none of it could be considered useful whatsoever.”
What I want to know here is whether it pays attention well to the header and judges accordingly. So, I again ask: “Please review the following document in terms of technology” and paste in the document. The reply is kind of spot on (and you can read the entire reply here):
“Review of “Really Bad Advice for SQL Server Administrators” (Technological Perspective)
This document is clearly a satirical piece aimed at highlighting poor practices in SQL Server administration. While humorous, its tone may be misinterpreted by novice readers as genuine advice. Below, I address key points raised in the article and provide a brief explanation of why each is bad practice, supported by technical reasoning.”
Ignoring that it says “I address key points” like the computer from 2001 A Space Odyssey, the replies in each section basically ignores the fact that this is satire. If it knows that, it the advice should show that. It should give advice how to keep the satirical edge, but not lead people astray.
For comparison, the row by row processing section now says:
“Row-by-Row Processing (RBAR) Instead of Set-Based Operations
Why This is Bad: Row-by-row processing is significantly slower than set-based operations, as SQL Server is optimized for set-based query execution.
Best Practice: Write queries that operate on sets of data rather than iterating through rows.”
Which is of course good advice, but it really isn’t a good section compared to what I was trying to do and include code. (Which it still doesn”t mention.)
Finally:
“Conclusion
The document is an effective humorous critique of common SQL Server mismanagement practices. However, it risks misinterpretation if taken literally. The importance of adhering to best practices—such as maintaining security, scalability, and performance—cannot be overstated.
Suggestion: Add a clear disclaimer at the beginning, emphasizing that the advice is intentionally bad and should not be followed. This will help ensure the content is understood as satire.”
I mean, it isn’t really “satire” per ce, it was framed as just bad advice that had been
Conclusion
So far, reasonably good. It is not nearly as good of an editor as a human would be, and it made a few mistakes. But I feel like I have answers my first question “Will it refute the advice clearly?” Not only did it refute the advice, but it called me out on my less than subtle advice.
I think it is clear that after I do this same test with Copilot, I need to refine the concept and be less overt with the advice and be more subtle with my errors. Code that fails for simple reasons. Compatibility levels, typos, invalid semicolons. Advice that is weak but true.
In fact, that is kind of why I did this series of tests to begin with. I commonly find that these tools I am testing say “this is a good paper” but I find some obvious concerns that are missed. I want to know that overall, at least, it is catching the gist of the paper and general correctness.
Either way, my concerns have somewhat been allayed. This tool definitely did not try to make it out like this was good advice, and it did a good job even with the reverse piece that said explicitly that this was bad advice.
I especially liked that is said “While humorous, its tone may be misinterpreted by novice readers as genuine advice.” I actually did not include the entire fake articles as posts for that very reason, and even zipped them up so they hopefully will not be read by the search engine.
Load comments